Skip to content

fix(ssh): retry a stalled SSH reconnect when the machine wakes or the network returns - #29237

Merged
nwparker merged 3 commits into
mainfrom
nwparker/ssh-wake-reconnect
Oct 16, 2026
Merged

nwparker merged 3 commits into
mainfrom
nwparker/ssh-wake-reconnect

Conversation

@nwparker

@nwparker nwparker commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​261 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​261
Prod 4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​130 $\color{#cf222e}{\Huge{\mathbf{−}}}$​31 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​99

ELI5

If your computer sleeps for a while, Orca can use up all of its automatic SSH reconnect attempts while the computer is still asleep. Once those attempts were used up, nothing tried again. You had to click Reconnect or restart Orca, even though a plain ssh to the same host worked fine. Now, when the computer wakes, the screen unlocks, or the network comes back, Orca retries right away, with a fresh set of attempts.

What Changed

Before

  • A host that ran out of reconnect attempts ("Max reconnection attempts reached", status reconnection-failed) stayed down until you reconnected it by hand. In [Bug]: SSH host stays in reconnection-failed after macOS sleep; no retry on wake #26888 it was still down 40 minutes after wake. On macOS, the attempts are used up during the short "dark wakes" while the lid is closed. Those dark wakes give no wake signal, so the attempts are gone before the real wake.
  • On wake, Orca checked only the managed-server tunnels it had a record of. A host waiting 30 s between attempts kept waiting. A connect attempt that started before sleep kept running against the old network route until its 30 s timeout.
  • When the window reported "network back" (the browser online event), that signal reached the remote-runtime connections but not SSH.

After

  • Wake (resume), screen unlock (unlock-screen), and the window's existing resume/online request (runtimeEnvironments:retryConnectionsNow) all run one function, retrySshConnectionsAfterNetworkReturn. That function restarts every SSH connection that is stalled in one of three ways: it gave up, it is waiting out a backoff delay, or a background attempt has been running for more than 5 s. The retry starts with a full set of attempts. The same function then re-checks the managed-server tunnels, as wake already did. A connection that still looks connected but whose route died is found by that check within about 10 s, rather than by keepalive.

Mechanism. SshConnection.retryAfterNetworkReturn() sits on top of the existing reconnect() path: it closes the transports, resets the backoff ladder, and makes one immediate attempt. If that attempt fails, the normal ladder takes over from the first step. SshConnectionManager.retryStalledReconnects() calls it for each connection. Healthy connections are left alone, and so are a first connect, a closed connection, and a connection waiting on a password or passphrase prompt (so an open dialog is never closed). Auth failures and permanent errors are also left alone.

Why

  • Reuse, not a new retry system. The wake and online signals were already wired up. They just never reached the SSH ladder. This change routes them there and reuses reconnect() and the managed-tunnel resume check.
  • Retry on a signal, not retry forever. Giving up after nine failed attempts stays the rule, so a host that is really gone still ends up in a clear state. Wake, unlock and network-back are the moments when a retry is likely to work, and they are exactly what the issue asks for.
  • The 5 s age guard. Main's own wake event and the window's wake request arrive within milliseconds of each other. Without the guard, the second signal would cancel the handshake the first one just started.
  • Not counting attempts made while asleep. Electron has no dark-wake event, so the attempts can't be paused while the machine sleeps. Starting a fresh set of attempts on the next real signal gets the same result.

Linked Issue

Fixes #26888
Fixes #21127

Not fixed here:

Visual Proof

N/A. Nothing new is drawn. The difference is that a host showing "reconnection-failed" or "reconnecting" after sleep comes back by itself after wake or when the network returns.

Testing

  • New src/main/ssh/ssh-connection-wake-retry.test.ts (fake timers, real SshConnection with the ssh2 mock). Covers: a ladder used up while asleep stays down for 10 minutes, then reconnects on the wake retry. A wake retry that fails gets a fresh ladder. A 30 s backoff is skipped. A hanging handshake older than 5 s is replaced, and a younger one is kept. Healthy, first-connect and closed connections are untouched. The manager restarts each stalled connection.

  • src/main/ipc/ssh-host-sleep-reconnect.test.ts: resume and unlock-screen restart stalled reconnects, the network-back path also re-checks tunnels, and the listeners go away once unregistered.

  • New src/main/ipc/runtime-environment-recovery-handler.test.ts: the window's resume/online request now also reaches SSH.

  • With the fix reverted, 10 of the 13 tests fail. The other three cover behavior that is unchanged.

  • pnpm test src/main/ssh/ssh-connection src/main/ssh/orcad-managed-tunnel src/main/ipc/ssh src/main/ipc/runtime-environment: 82 files, 675 tests pass. pnpm tc passes, and so does pnpm run check:code-quality:changed.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

Review

  • Cross-platform: resume fires on macOS, Windows and Linux. unlock-screen fires only on macOS and Windows, and on Linux the listener just never fires. The browser online event works everywhere.
  • SSH / execution host: this only redials the client-side transport. It never reports or ends remote work, and losing contact is never treated as a process exiting.
  • Wire compatibility: no protocol or IPC change. It reuses the existing runtimeEnvironments:retryConnectionsNow channel.
  • Performance: one check per connection per wake/unlock/online event. Only stalled connections redial.

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Overlap: none with #29210 (renderer pane recovery) or #29220 (move refusal). The work on serving-probe latency changes orcad-managed-serving.ts and the shared-control files, which this PR does not touch.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

📝 Walkthrough

SSH connections can now restart eligible stalled reconnect attempts after network return. The runtime-environment recovery handler invokes SSH retries, and registered resume and screen-unlock events use the same retry flow. When a user-data path is registered, that flow also probes managed Orca tunnels. Generation checks guard connection transport state from superseded attempts. Tests cover retry conditions, event handling, and unregister behavior.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: 🟡 Moderate · up to a0f59

A wake-triggered retry can use the wrong host-key verification settings if two connection attempts overlap. Guard the configuration assignment before merging.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 7 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: retrying stalled SSH reconnects after wake or network return.
Description check Passed The description is complete and relevant. It includes the required explanation, changed behavior, rationale, linked issues, visual proof, testing details, review considerations, and checklist status.
Linked Issues check Passed The PR meets the coding requirements in [#26888] and [#21127]. retrySshConnectionsAfterNetworkReturn handles resume, unlock-screen, and runtimeEnvironments:retryConnectionsNow. `retryStalledRe…
Out of Scope Changes check Passed The changes stay within the scope of [#26888] and [#21127]. The SSH retry method, connection-manager method, wake and unlock listeners, network-return integration, managed-tunnel recheck, generation g…
  • Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4e100ebf-ccc0-4e14-9ed7-d29bf3515dfb
📥 Commits

Reviewing files that changed from the base of the PR and between 1b1d09c and 4a295270257db957868201abfdbef35a5d554e56.

📒 Files selected for processing: 7
  • src/main/ipc/runtime-environment-recovery-handler.test.ts
  • src/main/ipc/runtime-environment-recovery-handler.ts
  • src/main/ipc/ssh-host-sleep-reconnect.test.ts
  • src/main/ipc/ssh-host-sleep-reconnect.ts
  • src/main/ssh/ssh-connection-manager.ts
  • src/main/ssh/ssh-connection-wake-retry.test.ts
  • src/main/ssh/ssh-connection.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

Comment thread src/main/ssh/ssh-connection.ts Resolved
nwparker and others added 3 commits October 15, 2026 16:17
… network returns

After sleep, an SSH host could sit in reconnection-failed forever: the retry
ladder spent its nine attempts while the Mac was asleep or in dark wakes, and
nothing restarted it. Resume only re-probed managed tunnels, and the window's
online signal never reached SSH at all.

- SshConnection.retryAfterNetworkReturn restarts a spent ladder, a pending
  backoff, or a background handshake older than 5s with a fresh budget.
- Resume, unlock-screen and the window's resume/online signal
  (runtimeEnvironments:retryConnectionsNow) all call it via the manager and
  re-probe managed tunnels, so a transport that still looks connected after a
  network change is caught by the probe instead of keepalive.

Co-Authored-By: Claude <noreply@anthropic.com>
…ement's transport

A wake retry can now replace an in-flight handshake. Closing a system-SSH
probe only signals the process, so its late close used to reset the
transport state the replacement had just set up, leaving a connected host
whose exec failed with 'Not connected'.

Co-Authored-By: Claude <noreply@anthropic.com>
…proxy process

Co-Authored-By: Claude <noreply@anthropic.com>
@nwparker
nwparker force-pushed the nwparker/ssh-wake-reconnect branch from 7cf4ce2 to a0f5930 Compare October 15, 2026 23:18

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

🟠 Major · Guard the configuration assignment against superseded attempts. · ssh-connection.ts:851

src/main/ssh/ssh-connection.ts:851
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Guard the configuration assignment against superseded attempts.

When this retry replaces an attempt waiting in resolveWithSshG, the old attempt can resume after the replacement assigns hostKeyResolvedConfig. The old attempt overwrites that field before its generation check. If the replacement is still checking security-key transport, doSsh2Connect can then use the old attempt’s strictHostKeyChecking or known-hosts configuration. Check the generation before assigning hostKeyResolvedConfig in attemptConnect, or keep the resolved configuration local to each attempt.


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 15db1282-88e0-4a16-99ef-a8fc131f17ce
📥 Commits

Reviewing files that changed from the base of the PR and between 4a295270257db957868201abfdbef35a5d554e56 and a0f5930.

📒 Files selected for processing: 2
  • src/main/ssh/ssh-connection-wake-retry.test.ts
  • src/main/ssh/ssh-connection.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 2 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant